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Dr.  Dexter  Fletcher 
US  Army  Research  Institute  for  the 
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Alexandria,  Va.  22314 


Dear  Dr.  Fletcher: 

Back  in  September  1979,  under  your  aegis,  HumRRO  conducted  a  workshop  on 
"Front-End  Analysis  to  Aid  Emerging  Training  Systems."  We  produced  a  work¬ 
shop  summary  report  by  Bob  Seidel  and  Hal  Wagner  in  February  1980,  but  never 
asked  for  "open  release  clearance." 

At  the  NSIA  conference  in  San  Diego  last  May,  you  invited  attendees  at  one 
of  your  seminars  to  contact  HumRRO  for  copies  of  this  workshop  summary.  I 
find  that  I  have  only  a  single  copy  left,  and  would  like  to  file  it  with  the 
Defense  Technical  Information  Service  (DTIC)  to  make  certain  that  it  will  re¬ 
main  available  to  requestors. 

If  you  have  no  objection,  we  will  be  happy  to  do  the  actual  filing  if  you  will 
be  kind  enough  to  approve  the  public  releasability  of  the  workshop  summary. 

Thank  you  for  the  kind  attention  I  am  certain  this  request  will  receive. 
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FRONT-END  ANALYSIS  TO  AID  EMERGING 
TRAINING  SYSTEMS 


Workshop  Summary 


I.  INTRODUCTION 

An  invitational  workshop  on  front-end  analysis  of  emerging  systems  was 
conducted  on  September  10-14,  1979,  by  HumRRO  under  Contract  MDA  903-78-C-0023 
to  the  Defense  Advanced  Research  Projects  Agency  (DARPA).  The  agenda  was 
developed  by  a  workshop  steering  committee  (see  Appendix  A  for  the  list  of 
Steering  Committee  members).  These  individuals  were  responsible  for  selecting 
the  issues  to  be  covered  at  the  workshop,  presenters  of  technical  papers, 
discussants,  and  other  workshop  participants.1 

This  report  summaries  the  major  points  covered  by  the  presentations  and 
subsequent  discussions.  In  addition,  the  final  section  lists  recommendations 
that  would  support  the  current  implementation  of  front-end  analyses,  and 
produce  a  comprehensive  R&D  program  to  improve  our  ability  to  conduct  such 
analyses  as  identified  by  workshop  participants. 


Definition  of  Front-End  Analysis  (FEA) 

The  definition  of  FEA  presented  below  resulted  from  the  workshop  and  follow¬ 
up  communications  with  attendees. 

Front-end  analysis  (FEA)  is  a  process  that  evaluates  requirements  for 
manpower,  personnel  and  training  (MP&T)  during  the  early  stages  of 
the  military  systems  acquisition  cycle.  Its  purpose  is  to  (1)  deter¬ 
mine  manpower,  personnel,  and  training  (MP&T)  requirements  under 
alternative  system  concepts  and  designs,  and  (2)  estimate  the  impact 
of  these  MP&T  requi rements  on  system  effectiveness  and  life-cycle 
costs.  Its  end-product  should  be  the  information  needed  to  assume 
that  effective  resources  (human,  equipment,  materiel)  will  be  avail¬ 
able  when  and  as  required  for  each  system  to  achieve  its  intended 
contribution  to  military  readiness  and  effectiveness. 


^A  list  of  all  workshop  participants  is  presented  as  Appendix  A. 
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Purpose  of  the  Workshop 

The  goals  of  the  workshop  were: 

1.  To  educate  the  manpower,  personnel,  and  training  (MP&T)  research 
and  development  community  in  the  process  of  systems  acquisition  and  of  their 
responsibilities  to  that  process. 

2.  To  exchange  information  between  and  within  the  Services  on  ongoing 
efforts  to  apply  FEA  to  systems  acquisition. 

3.  To  stimulate  development  of  an  informal  master  plan  for  R&D  to 
improve  our  ability  to  perform  FEA. 

4.  To  permit  informal  contacts  to  be  made  between  system  project 
officers,  MP&T  research  and  development  personnel,  and  individuals  at  various 
management  levels  in  the  civilian  and  military  communities. 

II.  PRESENTATIONS/ DISCUSS  IONS:  MAJOR  POINTS 

Some  of  the  major  points  derived  from  the  presentations  and  discussions 

2 

are  described  in  this  section.  Three  papers  provided  examples  of  different 
types  of  systems  acquisition:  an  existing  system,  an  emerging  system,  and  a 
"non-system"  (acquisition  of  equipment  for  which  there  is  no  Project 
Manager).  Four  papers  examined  the  methodologies  and  tools  that  exist  at 
present  for  aiding  front-end  analysis  (FEA):  the  Air  Force  Logistics 
Composite  Model  (LCOM),  the  Navy  HARDMAN  methodology,  the  Army  Personnel 
Affordability  program,  and  the  NAVAIR/NTEC  front-end  analysis  process.  These 
presentations  set  the  stage  for  subsequent  information  exchange  led  by 
designated  discussants  (see  Appendix  A  for  list  of  Presenters  and  Discussants). 

2 

Copies  of  the  full  papers  and/or  visuals  employed  at  this  workshop  are 
available  at  HumRRO  and  the  Cybernetics  Technology  Office,  DARPA. 
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A. 


EXISTING  SYSTEM:  AIR  FORCE 


•  The  F-16  aircraft  program  was  a  complex  procurement  involving  4400 
vendors  and  five  different  countries.  There  were  major  problems  in  deter¬ 
mining  training  requirements  and  in  obtaining  training  devices  in  a  timely 
manner.  These  problems  were  due,  in  part,  to  the  following: 

•  Only  3  months  (of  the  9-12  required)  were  allocated  for  a  task 
and  skill  analysis  for  this  complex  system. 

•  The  aircraft  configuration  was  continually  changing  during 
this  period. 

•  Although  deficiencies  in  the  prototype  aircraft  were  being 
corrected,  timely  delivery  of  data  regarding  these  changes 
did  not  occur. 

As  a  result,  interim  training  devices  were  inadequate.  This  led  to 
the  expenditure  of  enormous  numbers  of  hours  for  training  in  the  actual 
aircraft  which  would  have  been  accomplished  in  flight  simulators. 


•  The  lessons  learned  from  the  F-16  program  were  as  follows: 

•  One  cannot  create  the  training  devices/simulators  and  software 
until  the  aircraft  design  has  been  stabilized. 

•  Incorporating  weapon  system  changes  made  during  development 
into  training  devices/simulators  necessitates  a  delay  in 
deliveries  of  effective  flight  simulators. 

•  There  is  a  strong  need  for  interim  trainers  that  can  accommo¬ 
date  systems  changes  as  needed  until  it  is  feasible  to  deliver 
product  ion- type  simulators.  This  statement  recognizes  that 
the  goal  of  0MB  Circular  A-109  and  DoD  Directives  5000.1  and 
5000.2  areto  adequately  define  training  devices  at  the  front 
end.  However,  it  is  a  fact  of  life  that  the  training  devices 
will  lag  the  development  of  the  weapon  system. 


•  Problem  facing  system  program  managers.  With  the  occurrence  of  frequent 
changes  in  equipment  design,  how  can  we  get  data  on  actual  equipment  to 
simulator  designers  so  that  appropriately  designed  training  devices  and 
simulators  are  developed  and  fielded  in  a  timely  manner? 
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Discussion  Points 


•  Training  is  only  a  part  of  the  major  issue  which  is  life-cycle  support. 
Specific  readiness  issues  and  shortfalls  must  be  identified  early  on.  There 

is  a  need  to  determine  MP&T  requirements  as  part  of  the  acquisition  process 
at  DSARC  Milestone  0,  thereby  providing  more  adequate  FEA. 

•  0MB  Circular  A-109,  and  DoD  Directives  5000.1  and  5000.2,  which  have 
recently  been  revised,  are  written  to  put  the  force  of  DoD  policy  behind  the 
timely  application  of  FEA. 


Notes 


OMB  Circular  A-109.  This  circular  establishes  standard  policies  to  be  followed 
by  executive  branch  agencies  in  the  acquisition  process  including  matters 
related  to  manpower,  personnel,  and  training  (MP&T)  .  A-109  applies  to  all 

programs  of  acquisition  even  though  a  systan  may  be  one  of  a  kind  or  the  agency 
concerned  is  only  involved  in  developing  demonstration  hardware.  Needs  and 
program  objectives  are  henceforth  to  be  expressed  in  mission  terms,  and 
emphasis  is  given  to  initial  activities  of  the  system  acquisition  process  to 
permit  competitive  exploration  of  alternative  system  concepts  relevant  to  those 
mission  needs. 

DoD  Directive  5000.1  (currently  under  revision).  The  provisions  of  this 
directive  apply  to  the  acquisition  of  major  systems  within  the  Department  of 
Defense.  The  principles  in  this  directive  should  also  be  applied,  where 
appropriate,  to  the  acquisition  of  systems  not  designated  as  "major,"  i.e., 
less  than  $100  million.  Responsibility  for  management  of  system  acquisition 
programs  shall  be  decentralized  to  DoD  components,  except  for  decisions 
retained  by  the  Secretary  of  Defense. 

The  objectives  of  the  directive  apply  to  each  DoD  official  who  has  direct 
or  indirect  responsibility  for  the  acquisition  process.  These  officials  shall 
make  every  effort  to: 

1.  Ensure  that  an  effective  and  efficient  acquisition  strategy  is 
developed  and  tailored  to  each  system  acquisition  program. 

2.  Minimize  the  time  from  need  identification  to  introduction  of  each 
system  into  operational  use. 

3.  Achieve  the  most  cost-effective  balance  between  acquisition  and 
ownership  costs  and  system  effectiveness. 

4.  Correlate  individual  program  decisions  with  the  Planning,  Programming., 
and  Budgeting  System  (PPBS). 

5.  Maximize  collaboration  with  United  States  allies. 

6.  Integrate  support,  manpower,  and  related  concerns  into  the 
acquisition  process. 


DoD  Directive  5000.2.  The  purpose  of  this  directive  is  to  provide  proce¬ 
dures  for  DoD  use  in  implementation  of  DoD  Directive  5000.1.  It  applies  to 
all  major  systems  acquisition  of  the  Office  of  the  Secretary  of  Defense  (OSD), 
the  military  departments,  the  organization  of  the  Joint  Chiefs  of  Staff 
(OJCS),  and  the  Defense  Agencies. 

The  procedures  cover  major  system  designation  and  listings,  required  Milestone 
0  documentation  (the  Mission  Element  Needs  Statement),  the  role  and  procedures 
for  the  DSARC  (Defense  System  Acquisition  Review  Council).  Also  covered  are 
the  requirements  at  each  program  phase  for  analysis  and  documentation  of 
efforts  related  to  front-end  analysis  of  manpower,  training  and  logistics 
requirements.  Included  as  one  of  the  key  enclosures  of  this  directive  is  the 
outlines  for  the  Mission  Element  Needs  Statement  (MENS). 

(For  detailed  descriptions  and  copies  of  the  documents,  the  reader  is 
referred  to  Mr.  Russell  Shorey,  Office  of  the  Secretary  of  Defense,  Manpower, 
Reserve  Affairs  and  Logistics,  The  Pentagon,  Room  2B323,  Washington,  DC  20301.) 
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8.  EMERGING  SVSTEM:  MAW 


The  acquisition  of  major  weapon  systems  in  accordance  with  recent 
OMB  directives  (A-76  and  A-109)  has  placed  increasing  emphasis  on  using 
mission  analysis  for  early  guidance  of  system  development.  Implementing 
this  guidance  results  in  a  Mission  Element  Needs  Statement  (MENS)  as  a  basis 
for  the  acquisition  strategy  of  the  program  manager.  In  the  case  of  the  VTX 
Training  System  (VTXTS),  the  mission  analysis  becomes  part  of  the  front- 
end  analysis. 

The  need  for  the  VTX  Training  System  (VTXTS)  came  from  the  realization 
that  current  aircraft  used  by  the  Navy  for  Undergraduate  Jet  Pilot  training 
(T2C  and  TA4  aircraft)  were  approaching  the  end  of  their  life  cycles.  A  new 
training  system  is  required  to  address  learning  objectives  for  pilot  training 
in  the  1980's  and  beyond.  VTXTS  will  do  this  by  using  the  Instructional 
System  Development  methodology  and  findings  from  studies  of  aircraft  projected 
to  be  in  use  in  the  1990's  (F/A-18,  F14,  AGE,  EA6B,  etc.).  Hopefully,  the 
generic  requirements  generated  by  VTXTS  will  be  integrated  into  the  design 
and  acquisition  of  new  aircraft. 

In  summary: 

•  A  unique  feature  of  the  VTXTS  procurement  is  that  it  is  following 
the  new  directive  OMB  Circular  A-109  to  the  letter.  A  complete  training 
system,  not  just  a  new  airplane,  is  to  be  procured.  The  hope  in  the  VTXTS 
procurement  is  to  use  a  problem-solving  approach  to  training  that  will  take 
the  form  of  a  system  integration  effort. 

•  The  VTXTS  program  is  an  example  of  the  use  of  mission  analysis  to 
guide  a  weapon  systems  program  during  its  early  phases.  The  mission  analysis 
culminated  in  an  FEA  which  is  viewed  as  a  continuing  process  prior  to  the 


procurement  of  hardware.  Initial  baseline  data  will  be  refined  and  validated 
throughout  the  conceptual  and  demonstration  phases.  This  is  a  unique  start 
for  procurement  of  a  major  program. 

•  The  Navy  sees  a  unique  opportunity  in  the  VTXTS  to  address  the  design 
of  a  total  training  system  and  reap  the  benefits  of  having  all  major  elements 
of  the  system  optimized  to  fulfill  the  mission  requirement,  i.e.,  a  complete 
training  system  rather  than  the  procurement  of  a  new  training  airplane.  A 
major  issue  remains,  however,  and  that  is  whether  to  insist  on  early  design 
freeze  or  to  build  more  complicated  training  systems  that  are  sufficiently 
flexible  to  accommodate  design  changes. 

C.  NON-SYSTEM  PROCUREMENT:  ARM Y 

•  The  development  of  training  systems  in  the  Army  requires  interactions 
among  three  separate  commands:  TRADOC  (Training  and  Doctrine  Command) — 
responsible  for  the  generation  of  training  theory,  doctrine,  and  systems — 
initiates  the  training  system  requirement;  DARCOM  (Army  Materiel  Development 
and  Readiness  Command) — develops  the  Army's  equipment;  and  the  field 
commands — who  represent  all  the  soldiers  in  the  field  and,  therefore,  the 
ultimate  users. 

•  Non-system  devices  support  general  military  training,  two  or  more  systems, 
or  several  different  types  of  equipment.  PM  TRADE  is  the  primary  user  of 
front-end  analysis  (FEA)  for  non-system  acquisitions. 

•  Training  device  development  in  the  Army  is  complicated  by  the  fact  that 
a  number  of  major  subordinate  commands  under  DARCOM  may  act  as  individual 
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materiel  development  agencies,  procuring  devices  to  satisfy  their  own 

3 

training  needs  without  overall  coordination. 

•  Problems  in  the  Army's  procurement  process  for  non-system  trainers 
include  the  following: 

•  Early  front-end  analyses  have  not  been  performed. 

•  Training  devices  are  needed  before  the  total  training  systen  has 
been  determined. 

•  In  the  development  of  devices,  fidelity  is  emphasized  rather  than 
training  effectiveness. 

•  Neither  training  effectiveness  nor  the  value  of  training  effective¬ 
ness  has  been  quantified. 

Discussion  Points 

•  Problems  and  issues  described  above  regarding  the  Army's  procedure  for 
non-system  procurement  did  not  distinguish  between  existing  and  new  systems. 
With  regard  to  existing  systems,  "rear  end"  analyses  (i.e.,  using  field  data 
as  feedback  in  an  interactive  process)  help  in  the  development  of  training 
devices.  In  fact,  the  Instructional  Systems  Development  (ISD)  process  calls 
for  these  analyses  to  be  performed  and  their  results  to  be  used  as  input  to 
training  system  design  and  development. 

•  In  many  cases,  evaluation  criteria  have  not  been  established  and  so 
the  effectiveness  or  benefits  that  are  to  be  derived  from  new  training  devices 
cannot  be  properly  evaluated. 
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This  problem  was  resolved  subsequent  to  the  workshop.  The  Project  Manager 
for  Training  Devices  (PM  TRADE)  has  been  assigned  responsibility  to  review 
and  concur  with  all  training  device  plans. 


•  Existing  simulators  have  been  successfully  employed  for  training 
before  new  simulators  were  available.  One  use  was  that  of  low  fidelity,  part- 
task  trainers  for  air-to-air  refueling  training.  In  that  case,  the  Air  Force 
rented  time  on  existing  simulators.  One  must  try  to  use  existing  equipment 

or  existing  simulators  wherever  possible;  this  may  solve  the  problem  discussed 
earlier,  namely,  how  to  make  up  for  the  lack  of  availability  of  trainers  at 
the  time  the  first  aircraft  is  delivered. 

•  There  is  a  great  need  for  identifying  training  and  human  factors 
requirements  as  early  as  possible  in  the  system's  design  and  development  cycle — 
that  is,  prior  to  Milestone  1.  Unfortunately,  solutions  to  this  problem  are 
currently  lacking. 

•  Training  people  have  not  been  sensitive  to  trends  in  weapon  systems 
design,  and  they  have  assumed  that  weapon  system  designs  are  unchangeable. 

•  The  principal  issue  in  front-end  analysis  is  not  training  but  job 
design.  Skill  level  estimates  should  be  determined  from  job  design  data 

and  then  checked  against  a  skills  inventory.  There  is  a  need  to  develop  skills 
inventories  and  a  system  to  manage  such  skill  inventories  once  they  are 
developed. 

V.  CONCEPTUAL  LEVEL :  LCOU  (AIR  FORCE ) 

•  LCOM  stands  for  "Logistics  Composite  Model,"  developed  by  The  Rand 
Corporation  for  the  Air  Force.  It  gives  point  estimates  of  needs  and  refers 
to  maintenance  manpower  modeling.  In  the  Air  Force,  LCOM  is  the  major  one 
of  10  separate  FEA  activities  going  on  simultaneously.  This  conceptual 
framework  draws  upon  additional  techniques  such  as  the  following: 

•  Human  Engineering  analyses 

•  Integrated  logistics  system  (TLS) 

•  Product  performance  feedback  system 


•  Instructional  systems  development  (ISD) 

•  Job  guide  development 

0  Human  resources  design  option  decision  trees 

•  System  ownership  cost  models 

•  Consolidated  data  base 

•  Occupational  analysis 

•  Logistics  and  human  resources  are  tightly  interwoven  and  the  LCOM  model 
provides  a  basis  for  determining  life-cycle  costing  and  tradeoffs  among  the 
various  human  or  equipment  resources.  It  draws  upon  task  analysis  information, 
job  guides,  simulation  of  failure  rate,  flying  hours  involved,  and,  in  general, 
draws  upon  many  of  the  existing  and  emerging  technologies  in  order  to  perform 
an  appropriate  man-machine  analysis  for  life-cycle  costing. 

•  Many  emerging  technologies  should  be  taken  advantage  of  in  performing 
front-end  analyses  (FEA).  One  such  model.  Life  Cycle  Costing  Model  (LCCM) , 
runs  on  a  large  computer  and  is  used  to  identify  "hot"  spots  (why  so  many 
people,  needs  for  spares,  etc.).  However,  it  also  takes  hours  to  run. 

Briefer,  more  efficient  versions  of  this  model  are  needed. 

•  The  data  base  for  given  equipment  has  to  exist  before  these  computer 
models  can  be  run.  This  should  not  be  too  difficult  because  75%  of  "new" 
hardware  systems  already  exist  and  just  need  to  be  upgraded,  rather  than 
created  as  totally  new  systems. 

Discussion  Points 

•  Even  in  new  systems,  there  is  a  dependence  on  laboratory  data  from 
the  manufacturers  in  order  to  get  reliability  and  maintainability  information. 

•  Tools,  as  represented  by  LCOM,  are  available  for  FEA  but  one  problem 
is  that  the  military  does  not  follow  these  "models."  These  models  should  be 
consolidated,  they  should  be  validated  and  improved,  and  a  methodology  for 
selecting  them  should  be  developed. 
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E.  EQUIPMENT  INTENSIVE  LEVEL:  HARDMAN  (NAW) 


•  The  HARDMAN  project  (Hardware  vs.  Manpower)  was  initiated  by  the  Navy  to 
provide  a  methodology  to  adequately  consider  the  implications  resulting  from 
weapon  system  design  decisions  upon  manpower,  personnel,  and  training  (MP&T). 
In  1977,  the  HARDMAN  study  found  that  inadequate  attention  was  being  paid  to 
manpower  costs  throughout  the  weapon  system  acquisition  process.  Most 
manpower  and  training  plans  occur  late  in  the  cycle  and  require  costly  repro- 
granming  because  of  a  lack  of  incentives  for  program  managers  to  concentrate 
on  manpower  costs,  lack  of  MP&T  policy,  and  lack  of  assessment  tendency  for 
life-cycle  support. 

•  The  Navy  has  tried  to  integrate  MP&T  into  the  weapon  system  design  and 
acquisition  process  by  means  of  HARDMAN  which  has  four  main  objectives: 

•  Institute  concise  procedures  to  address  MP&T  requirements  con¬ 
sistent  with  the  direction  of  higher  authority. 

•  Provide  the  means  of  compliance  with  policy  and  procedures  to 
the  acquisition  community. 

•  Develop  tools  and  methodologies  to  assist  program  managers 
when  considering  impact  of  system  design  and  the  acquisition 
process  on  MP&T. 

•  Provide  the  Chief  of  Naval  Operations  an  assessment  of  MP&T 
supportability,  i.e. ,  affordability  and  attainability  of  each 
new  acquisition  before  major  decisions  and  resource  allocations 
are  made. 

•  Today,  the  Navy  provides  the  inputs  for  MP&T  after  DSARC  Milestone  2, 
and  in  many  instances  this  is  too  late.  This  is  definitely  too  late  in  the 
acquisition  process  to  allow  for  any  man-machine  tradeoffs.  The  hope  is  that 
with  the  use  of  HARDMAN  methodology  to  identify  the  manpower  constraints, 

and  present  plans  for  addressing  productivity  changes,  these  inputs  can  be 


made  at  Milestone  1.  By  Milestone  2,  the  tradeoff  analyses,  vis-a-vis, 
manpower  and  design  alternatives  and  the  rationales  thereof,  will  have  been 
performed  (and  will  continue  through  Milestone  3). 

•  In  summary,  there  are  six  steps  to  the  HARDMAN  methodology: 

•  Establish  a  consolidated  data  base. 

•  Perform  a  manpower  requirements  analysis  (e.g. ,  indicate  the 
total  number  of  personnel  required  far  the  weapons  system. 

•  Perform  a  training  requirements  analysis. 

•  Perform  a  personnel  requirements  analysis  (i.e.,  a  breakdown 
of  needs  by  ratings  or  "faces"). 

•  Provide  an  impact  analysis. 

•  Determine  the  potential  tradeoff  areas  and  i te rate  the 
methodology. 

All  of  this  must  happen  prior  to  DSARC  Milestone  2.  As  a  goal,  HARDMAN  should 
operate  in  the  design  phase  with  a  paper  system  in  order  to  integrate  man¬ 
power,  personnel,  and  training  (MP&T)  requirements  into  the  weapon  system 
acquisition  process.  The  methodology  can  be  used  throughout  the  acquisition 
process  to  evaluate  man-machine  tradeoffs,  various  maintenance  concepts,  new 
learning  techniques  or  other  types  of  alternatives  submitted  to  the  program 
manager. 

Discussion  Points 

•  One  principal  problem  is  how  to  get  a  Program  Manager  to  spend 
money  early  in  the  decision  process  for  a  HARDMAN-type  analysis  when  he  is 
trying  to  hold  down  his  costs.  A  possible  answer  is  that  the  Program  Manager 
should  have  support  to  implement  these  analyses  from  Congress,  as  well  as 

from  within  the  Navy,  and  at  the  OSD  level  within  DoD.  As  it  is.  Program  Managers 
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typically  allocate  funds  to  MP&T  support  for  later  use  as  discretionary  funds 
for  spares  and  engineering  changes. 

•  Another  problem  is  the  practical  one  of  having  the  ability  to  assess 
supportability  before  acquisition  decisions  are  made.  The  Chief  of  Naval 
Operations  appears  to  be  the  only  one  who  can  make  this  assessment  (as 
currently  in  the  VTXTS  program). 

F.  LABOR  INTENSIVE  LEVEL:  AFFORDABILITY  (ARM/) 

•  Personnel  affordability  is  a  research  program  to  develop  specific 
techniques  to  enable  tradeoffs  with  respect  to  human  factors,  personnel, 
training,  and  hardware  requirements  early  in  the  life  cycle  of  emerging  weapon 
systems.  Two  relevant  AR1  research  efforts  are:  Cost  and  Training  Effective¬ 
ness  Analysis  (CTEA),  and  the  Early  Training  Estimation  System  (ETES).  CTEA 
concerns  the  selection  and  analysis  of  training  programs  within  the  system  life 
cycle.  ETES  concerns  the  development  of  a  task  data  base  during  the  early 
conceptual  development  stages  of  a  systen. 

•  In  order  to  provide  an  accurate  assessment  of  emerging  training  systems, 
there  is  a  need  to  develop  support  information  and  technologies  in  four  areas: 

•  Task  definition,  data  base  structure,  and  standardization,  both 
within  the  Army  and  across  DoD.  Task  formats  need  to  be  estab¬ 
lished  that  have  task  information  relevant  to  automated  training 
development  aids  and  to  the  software  application  requirements 
over  the  entire  instruction  development  cycle. 

•  Research  and  development  dealing  with  emerging  techniques  for 
training  development  have  to  be  coordinated  under  a  unified  Army 
utilization  plan. 

•  Methods  must  be  developed  for  comparison  of  derived  skills  and 
knowledges  across  Military  Occupational  Specialties  (MOSs). 

•  The  hardware  design  process  must  structure  equipment  descrip¬ 
tions  so  as  to  lead  directly  to  training- compatible  task  descrip¬ 
tions.  This  should  be  accomplished  early  in  the  conceptual  stage. 


•  A  series  of  tradeoff  decisions  must  be  made  early  encompassing  hardware 
development  and  MP&T.  Appropriate  tradeoff  analysis  aids  have  to  be  developed 
and  refined.  In  performing  training  tradeoff  analyses,  there  is  a  need  to 
determine  what  the  local  resource  impacts  are  from  the  projected  program. 

This  means  utilizing  existing  course  materials  wherever  feasible,  utilizing 
automated  decision  aids  within  the  schools  to  reduce  the  training  developer 
workloads,  and  determining  a  common  descriptive  format  to  link  developing 
concepts  with  training  needs. 

•  In  sum,  personnel  affordability  involves  tradeoff  decisions  stemming 
from  hardware  development  and  the  status  of  the  manpower  pool  to  implement 
that  hardware.  The  input  areas  are  human  factors,  training,  personnel,  and 
the  attendant  costs  related  to  these  efforts.  The  activities  involved 
include  relating  aptitude  measures  to  task  performance,  evaluating  the 
training  programs  as  a  function  of  the  life  cycle  management  model  stage,  and 
taking  into  account  the  transition  between  hardware  concepts  and  task 
definitions  needed  to  feed  automated  training  aids. 

Discussion  Points 

•  One  needs  to  consider  all  costs  and  impacts  of  alternative  training 
systems  during  the  early  stages  of  the  system  acquisition  process.  Such  cost 
and  effectiveness  data  are  needed  in  all  Services.  One  problem  to  be  solved 
concerns  the  source  of  funds  needed  to  do  extensive  CTEA  studies. 


G.  ON- GOING  FRONT-  END  ANAiySIS:  NT  EC  (NAl/y) 


•  The  principal  focus  of  front-end  analysis  (FEA)  at  the  Naval  Training 
Equipment  Center  (NTEC)  is  a  set  of  integrated  activities  ultimately  directed 
at  providing  choices  among  alternative  instructional  regimens.  FEA  at  NTEC 
has  five  steps  which  form  its  principal  set  of  activities. 

•  Specification  of  training  requirements 

•  Establishment  of  instructional  alternatives 

•  Cost  analysis  of  each  alternative 

•  Evaluation  of  the  effectiveness  of  each  alternative 

•  Selection  of  the  most  cost-effective  alternative 

An  additional  characteristic  of  this  version  of  FEA  is  the  iterative  process 
needed  to  refine  the  data  throughout  the  system's  entire  life  cycle. 

•  In  establishing  training  requirements  for  a  system,  one  must  distinguish 
between  the  constraint-free  or  ideal  environment  versus  that  of  the  real  world 
which  has  a  number  of  constraints  placed  upon  it.  The  NAVAIR/NTEC  ISD  model 
was  described  as  a  pragmatic  ideal. 

•  One  of  the  analytic  approaches  used  at  NTEC  is  to  develop  a  generic  data 
base  of  tasks.  The  steps  in  this  process  are:  work  from  existing  systems' 
task  inventories  to  establish  a  common  or  generic  data  base  of  tasks;  develop, 
from  these  data,  generalized  training  requirements  for  a  given  class  of 
systens;  and  also  provide,  as  a  result  of  this  generic  data  base  of  tasks,  a 
baseline  for  future  systems  analysis. 

•  As  one  is  able  to  increase  the  specificity  of  the  generic  task  data  base 
and  the  specificity  of  the  levels  of  instructional  regimens,  then  one  can 
continue  to  refine  the  entire  process  of  FEA. 

•  A  major  problem  is  the  lack  of  an  approach  for  specifying  training  media 
alternatives  and  iteratively  improving  upon  those  choices  in  FEA. 
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Discussion  Points 


•  In  ISD  we  do  not  describe  adequately,  or  in  a  timely  manner,  how  to 
determine  and  select  simulator  cost  and  fidelity  characteristics  needed  to 
make  simulators  effective  for  training. 

•  One  problem  is  obtaining  user  acceptance  without  assuming  that  the 
type  of  training  equipment  specified  by  the  user  (often  a  high  fidelity 
simulator)  is  necessarily  most  cost-effective. 

Discussion  Points  Concerning  All  Service  Methodologies 

•  There  are  major  differences  among  the  Service  efforts  in  maturity 
of  effort,  level  of  detail,  and  applicability. 

•  There  is  a  substantial  need  for  standardized  definitions  of  such 
terms  as  task  analysis,  measures  of  effectiveness,  measures  of  cost,  and 
transfer  of  training. 

•  The  Services  have  substantial  information  about  FEA.  This  information 
should  be  compiled  and  made  more  evaiiable. 

•  No  FEA' s  have  been  validated. 

•  Managerial  and  institutional  aspects  of  FEA  deserve  more  attention. 

The  incentives  for  Program  Managers  to  identify  roles  for  R&D  managers  and 
training  developers  should  be  recognized  and  strengthened. 
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III.  WORKING  GROUP  SESSIONS 

The  participants  were  divided  into  three  working  groups,  each  of  which 
was  to  discuss  the  same  six  issues,  and  then  report  their  conclusions  to 
the  entire  workshop.  These  six  issues  were: 

•  Characterize  what  an  effective,  practicable  FEA  technology  should 
be — including  a  definition  of  what  FEA  is,  what  the  essential 
components  of  FEA  are,  and  how  results  of  FEA  should  be  reported. 

•  List  specific  technologies  we  have  that  can  be  used  now  to 
contribute  to  the  FEA  process. 

•  List  specific  results/accomplishments  that  the  MP&T  community 
ought  to  achieve  in  the  next  2  years  in  support  of  FEA. 

•  List  specific  results/accomplishments  that  the  MP&T  community 
ought  to  achieve  in  the  next  5-7  years  in  support  of  FEA. 

0  Recommend  specific  follow-on  actions  that  should  occur  to  produce 
value  from  this  Workshop. 

•  Recommend  specific  actions  that  should  occur  to  establish  FEA  as 
an  essential  component  of  the  systems  acquisition  process. 

Working  group  summaries  were  prepared  and  distributed  at  the  Workshop.  (The 

working  group  leaders,  whose  significant  contributions  are  acknowledged  and 

appreciated,  are  listed  in  Appendix  A.  The  complete  working  group  summaries 

are  reported  in  Appendix  B.) 

The  first  issue  concerned  defining  front-end  analysis,  and  the  results 
of  these  discussions  were  incorporated  in  the  definition  presented  on  page  1. 
Similarly,  the  recommendations  for  further  research  are  summarized  in 
Section  IV,  pages  24-26. 

I .  Define  Front-End  Analysis  (FEA). 

A  temporal  perspective  needs  to  be  added  to  the  definition  of  FEA.  FEA 
occurs  up  through,  but  not  later  than.  Milestone  2  in  the  acquisition  process. 
The  initial  stage  of  FEA  is  a  conceptual  study  designed  principally  to 
influence  the  design  of  those  aspects  of  the  weapon  system  which  affect 
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contribute  to 


manpower,  personnel  and  training  on  a  life-cycle  basis  (i.e., 
weapon  system  concepts  and  requirements,  to  the  support  concepts,  to  personnel 
performance  requirements,  and  to  the  training  subsystem  requirements).  The 
second  stage  of  FEA  is  designed  principally  to  influence  the  development 
of  the  training  subsystem,  based  on  such  models  as  ISD.  The  products  at  the 
end  of  both  FEA  stages  will  usually  be  statements  of  requirements.  These 
include:  manpower  requirements  (including  numbers  and  spaces);  personnel 
requirements  (specific  types  of  people  and  skills);  training  requirements; 
selection  and  recruitment;  and  costs  (including  costs  of  acquisition,  recurring 
costs,  and  facilities).  The  specificity  and  depth  of  the  front-end  analyses 
(FEA)  will  depend  upon  when  in  the  weapon  system  development  cycle  the  study 
is  done.  In  the  earliest  conceptual  stages,  the  analyses  will  be  approximate 
and  general.  With  iteration,  details  and  depth  will  evolve,  but  all  analyses 
should  be  completed  by  DSARC  Milestone  2. 

The  results  of  the  FEA  should  be  reported  to  the  Program  Manager  in  terms 
that  would  permit  a  choice  of  alternative  systems  depending  upon  relative 
costs  and  risks  and  should  include  the  constraints  and  tradeoffs  operating  in 
each  alternative  concept  or  design  proposed. 

2.  List  the  specific  technologies  we  now  have  that  can  be  used  to 
contribute  to  the  FEA  process. 

Five  specific  technologies  emerged  from  these  discussions: 

a.  Compute rs.  Data  bases,  such  as  job/task/ skill  inventories,  which 
exist  now  can  be  placed  in  a  computerized  form.  Also,  there  are  computer-based 
simulation  techniques  and  manpower  models  currently  in  use.  These  computer 
simulations  or  models  need  to  be  improved  and  validated  to  increase  their 
predictive  capabilities. 
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b.  Costing  Model s/Techniques.  Costing  models  and  techniques  exist  which 
can  enhance  FEA.  For  example,  COEA  and  CTEA  models  have  be*>n  developed  and 
applied  in  the  Army.  There  are  also  econometric  models,  accounting/budget 
models,  and  life  cycle  cost  models  which  can  be  used  during  the  conduct  of 


c.  Training  System  Development  Models/Aids  (TRAMOD) .  Models  such  as 
1SD  need  refinement  and  amplification.  ISD  falls  short  in  describing  the 
training  device/slmuiator  selection  process,  but  this  could  be  improved. 

Also,  there  are  tools,  such  as  TRAMOD,  which  help  determine  training  require¬ 
ments  from  the  results  of  a  task  analysis. 

d.  Testing  and  Evaluation.  It  was  suggested  that  FEA  could  benefit 
from  comparability  analyses  of  new  systems  with  old  by  examining  historical 
data  of  various  systems.  These  data  can  be  obtained  from  tests  and  evalua¬ 
tions  such  as  design  tests,  operational  tests,  and  unit  readiness  measures, 
all  of  which  could  contribute  to  FEA. 


e.  Job  Aids.  Existing  guides  and  handbooks  can  be  improved  to  provide 
better  information  for  both  engineers/designers  and  Program  Managers  throughout 
the  entire  acquisition  process. 


3.  List  the  specific  resul ts/accompl  ishments  that  the  Manpower,  Personnel 
and  Training  (MP&T)  research  community  ought  to  achieve  in  the  next 
two  years  in  support  of  FEA. 


There  is  a  need  to  develop  better  cost  data  and  better  cost  models,  with 
examples  and  guidance  on  how  to  use  the  data  and  models,  particularly  as  they 
refer  to  critical  factors  in  FEA. 

Also  needed  are  operational  definitions  of  the  inputs,  processes,  and 
outputs  of  FEA.  A  common  understanding  of  FEA  terminology  will  require  a 


set  of  operationally  defined  measures.  If  this  is  accomplished  properly,  all 
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Services  will  be  able  to  use  common  data  bases  in  their  analyses.  This  will 
also  permit  using  cost-effectiveness  data  to  study  alternative  system  feasi¬ 
bility,  as  well  as  being  able  to  "fine  tune"  systems.  Standardization  and 
operationalization  will  also  permit  better  integration  of  established  ISD 
methodology  within  the  total  FEA  process.  It  will  allow  the  development  of 
a  more  general  metric  for  training  requirements  than  is  currently  in  use. 

It  will  also  permit  the  identification  of  critical  factors  (i.e.,  the 
"drivers,"  and  "big  payoff"  areas,  etc.)  which  are  the  important  forces  behind 
the  results  of  typical  FEAs.  We  will  also  be  able  to  assess  the  state  of  the 
data  bases — identifying  gaps  in  the  data  base  and  problems  in  its  utilization. 

One  major  gap  in  the  data  base  is  information  regarding  maintenance 
training,  maintenance  performance,  maintainability,  etc.  For  example,  in 
existing  systems,  a  major  improvement  in  maintenance  performance  is  believed 
to  be  feasible  and  should  be  a  goal  of  FEA.  This  could  be  accomplished  if 
there  was  a  concurrent  development  of  guidelines  for  quality  assurance  procedures 
on  FEA  components,  and  methods  for  precise  specification  of  system  performance 
factors  associated  with  the  human  component.  The  latter  will  probably  involve 
new  methods  for  skill  level  determination  and  new  methods  by  which  users  can 
better  identify  training  needs. 

4.  List  the  specific  resul ts/accompl ishments  that  the  MPT  research 
community  ought  to  achieve  in  the  next  5  to  7  years  in 
support  of  FEA. 

We  need  better  prediction  models  for  relating  options  in  early  weapon 
system  design  to  MP&T;  work  should  continue  on  aggregate  manpower  models. 

There  should  be  a  continuing  development  of  computerized  models  to  assist 
in  the  system  design  relative  to  MP&T.  More  work  is  also  needed  to  assess 
the  validity  of  MP&T  predictions  so  that  further  modifications  of  these  models 
can  be  accomplished. 
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There  should  be  a  continuous  identification  process  of  user  training 
needs.  In  this  regard, there  needs  to  be  a  skills  inventory  and  population 
quality  census  (e.g.,  an  identification  of  the  distribution  of  aptitudes 
within  the  manpower  pool,  such  as  project  TALENT). 

In  order  to  implement  all  of  the  requirements  noted  above,  there  is  a 
need  to  develop,  or  continue  to  refine,  a  computerized  management  informa¬ 
tion  system  (perhaps  implemented  on  the  ARPANET).  This  should  involve  the 
three  Services  so  that  they  all  could  provide  and  make  available  data 
required  for  FEA. 

Job  aids  in  the  area  of  MP&T  for  Program  Managers  need  to  be  developed 
such  as  handbooks  and  guides.  In  addition,  efforts  are  needed  to  improve 
awareness  of  and  use  of  MPT  problem-solving  capabilities  among  management 
personnel,  designers,  and  the  R&D  community. 

5.  Recommend  specific  follow-on  actions  to  produce  value 
from  the  workshop. 

Involve  the  Program  Managers  more  fully  in  the  entire  FEA  process.  More 
Program  Managers  need  to  be  involved  in  a  follow-up  workshop  of  this  kind. 

Also,  a  curriculum  on  FEA  should  be  developed  and  implemented  for  Program 
Managers  (possibly  at  the  Defense  Systems  Management  College,  Ft.  Belvoir, 
Virginia) . 

More  dialogue  between  management  personnel,  users,  and  designers  is 
needed  to  insure  that  the  results  of  FEAs  would  be  made  available  in  time  to 
effect  the  design  of  new  or  emerging  weapon  systems. 

The  results  of  the  workshop  need  to  be  integrated  into  a  presentation 
and  a  plan  for  use  by  R&D  management  personnel  to  publicize  the  needs  for  and 
capabilities  to  perform  FEA  in  support  of  weapon  system  development.  A  tri- 
Service  list  of  available  technologies  should  be  assembled  in  some  standardized 
form  for  distribution  throughout  DoD. 
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6.  Recommend  specific  actions  that  should  occur  to  help  ensure 
establishment  of  FEA  as  an  essential  component  of  the 
systems  acquisition  process. 

There  is  a  need  to  establish  an  action  office  or  advocate  In  the  DoD  to 
help  make  the  FEA  process  work.  In  order  to  establish  this  advocacy  role, 
RDT&E  funding  should  be  made  available  to  Insure  that  data  and  findings 
with  respect  to  FEA  are  used  in  activities  related  to  the  DSARC  process.  As 
part  of  this  effort,  there  is  a  need  to  advise  and  inform  Program  Managers 
and  all  other  decision  makers  about  the  impact  that  FEA  on  MP&T  can  have  on 
total  DoD  requirements. 

There  is  a  need  to  assure  stability  of  personnel  as  it  applies  to  contin¬ 
uity  of  the  FEA  process.  If  there  is  too  much  rotation  of  key  personnel,  then 
FEA  started  by  some  Individuals  will  not  be  continued  in  the  same  way  by 
others.  In  general,  stability  in  assignments  is  needed  for  personnel  involved 
in  large  system  acquisitions. 

FEA  can  become  an  essential  component  of  the  systems  acquisition  process 
if  existing  regulations  such  as  OMB  Circular  A-109,  DoD  Directives  5000.1  and 
5000.2  are  implemented  and  enforced.  In  this  regard,  it  was  recommended  that 
FEA  job  aids  be  developed  and  provided  to  Program  Managers.  In  addition, 
requirements  for  accountability  on  the  part  of  Program  Managers  are  necessary 
to  ensure  FEA  implementation,  as  well  as  incentives  for  contractors  to  perform 
adequate  FEAs. 

Concluding  Discussion  Points--Fu11  Workshop 

9  The  need  to  identify  and  compile  existing  FEA  work  in  the  Services  was 
reiterated  and  emphasized  as  was  the  need  to  validate  existing  FEA's.  There 
is  a  neea  to  improve  low  cost  models  with  major  attention  to  the  amount  of 
detail  required. 
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•  FEA  should  be  driven  by  threat  and  need.  We  must  be  able  to  assess 
the  current  position. 

•  Low  cost  simulators  deserve  high  priority.  In  general,  we  need  better 
kinds  of  simulation. 

•  Validation  of  FEA  will  require  validation  of  personnel  selection  and 
classification  procedures.  An  aptitude  and  skills  inventory  should  be 
developed  and  maintained. 

•  The  need  to  standardize  FEA  terms  was  reiterated  and  emphasized. 

•  There  should  be  an  OSD  action  officer  for  FEA. 

•  A  briefing  on  FEA  should  be  developed  for  OSD  managers.  Congress, 
Program  Managers,  and  training  developers. 

•  Authorizing  directives  for  FEA  now  exist.  They  should  be  implemented 
and  followed.  The  most  important  of  these  are  Directives  5000.1,  5000.2  and 
5000.39.  The  MENS  (Mission  Element  Needs  Statement),  the  Decision  Coordinating 
Paper  (which  serves  as  a  charter  for  the  Program  Manager),  and  the  Integrated 
Program  Summary,  Annex  D  (which  is  the  manpower  annex),  the  manpower  require¬ 
ments  document,  the  logistic  support  analysis — all  require  documentation  of 

an  FEA  in  one  form  or  another.  The  demand  for  a  structured  analysis  exists. 

The  problem  Is  to  produce  quality  analysis. 


23 


IV.  RECOMMENDATIONS 


As  a  result  of  the  working  group  discussions,  recommendations  for  future 
actions  regarding  FEA  were  generated.  These  were  subsequently  organized  into 
15  recommendations  and  mailed  to  the  workshop  participants.  Twelve  respondents 
provided  additional  information  concerning  the  relative  importance  of  these 
recommendations.  The  highest  priority  was  given  to  the  recommendation  for  a 
plan  or  a  "roadmap"  to  develop  a  standardized  technology  for  FEA  with  heavy 
emphasis  on  validation.  Similarly,  a  large  number  of  respondents  thought  that 
there  is  a  need  to  specify  the  detail  required  for  analytical  purposes  in 
the  MP&T  area  for  FEA  at  Milestones  0,  1  and  2.  On  the  other  hand,  the 
respondents  agreed  that  it  would  be  relatively  unimportant  to  obtain  FEA 
information  from  other  countries  for  purpose  of  comparison.  The  remaining 
recommendations  were  viewed  to  be  appropriate  but  with  not  much  agreement  on 
their  relative  importance.  All  15  recommendations  have  been  categorized  below 
in  terms  of  whether  they  concern  matters  of  policy  or  the  conduct  of  future 
R&D. 

Recommendations:  Pol  icy  Actions 

1.  Produce  a  plan,  or  "roadmap,"  that  prioritizes  what  must  be  done 
to  produce  a  usable,  standardized  technology  for  front-end  analysis. 

2.  Establish  uniformity  in  existing  manpower,  personnel,  and  training 
(MP&T)  guidance,  reporting,  and  data  base  collection  and  classification. 

The  first  step  would  be  to  collect  existing  Service  and  DoD  documents  (e. g. , 
the  TRADOC  training  affordability  handbook,  a  data-task  analysis  handbook, 

a  glossary  of  standardized  FEA  terras,  universal  computer  program  packages 
covering  frequently  used  FEA  techniques,  etc.).  The  second  step  would  be  to 
develop  and  publish  understandable  pamphlets  and  workbooks  for  FEA  aimed  at 
practitioners  with  pointers  to  more  detailed  references  ("cookbook"). 


3.  Begin  to  institute  the  raanagement-user-designer  dialogue,  for 
example,  through  follow-on  workshops  that  would  include  the  Program  Managers 
as  primary  participants  (e.g. ,  conduct  such  a  follow-on  workshop  in  about 

8  months). 

4.  Identify  who  should  have  the  ressponsibility  to  perform  the  follow¬ 
up  R&D  and  management  actions  noted  herein  (other  than  DARPA).  For  example, 
identify  an  OSD  action  officer,  supported  perhaps  by  a  small  working  group, 
with  budget  and  authority  to  perform  these  actions. 

5.  Develop  and  conduct  briefings/presentations  to  top  DoD  and  Congresssional 
officials  on  the  need  for  and  potential  benefits  of  FEA  and  a  description  of  a 
program  to  achieve  these  goals. 

Recommendations:  R&D  Actions 

1.  Obtain  data  on  how  selected  major  training  systems  were  developed 
to  identify  critical  issues  that  affected  design,  availability,  and  cost  of 
trainers  and  training  (e.g.,  the  F-16,  the  B-52  refueling  trainer  fly-off,  etc.). 

2.  Identify  areas  where  more  data  and/or  information  (to  aid  in  FEA  or 
to  guide  the  conduct  of  FEA)  are  needed.  Sources  for  this  might  be  case 
studies  of  MP&T  factors  in  recent  DSARC  reviews  (e.g.,  the  XM-1,  the  ES-111, 
ROLAND,  LAMPS,  etc.). 

3.  Describe  the  current  state-of-the-art  in  FEA  (generate  an  annotated 
list  of  available  technologies),  and  define  the  requirement  for  new  R&D.  One 
primary  source  could  be  findings  of  preconceptual  studies  done  on  the  VTXTS. 

4.  Generate  a  tri-Service  list  of  on-going  activities  concerned  with 
the  development  of  FEA, 

5.  Establish  criteria  by  which  to  evaluate  the  capabilities  of  existing 
engineering  simulators  to  validate  training  requirements  early  (especially 
prior  to  delivery  of  complex  and  costly  simulators). 
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6.  Simplify  the  LCOM  type  models  in  initial  analyses  of  MP&T  and 


develop  other  prediction  techniques. 

7.  Specify  the  amoupt  of  detail  required  for  analytical  purposes  In 

the  MP&T  area  at  Milestones  0,  1  and  2.  For  example,  develop  methods  for  more 
precise  quantitative  specification  of  system  performance  factors  associated 
with  the  human  component.  A  second  example  would  be  to  develop  useful  and 
agreed-upon  operational  definitions  and  measures  of  training  system  costs  and 
effectiveness  suitable  for  varied  stages  of  FEA  application.  Third,  develop 
a  methodology  for  simple,  low-cost  means  of  generating  audit  trails  during  the 
course  of  FEA.  Fourth,  prepare  a  guide  for  Program  Managers  which  would 
include  data-item  descriptions  for  data  impacting  MP&T  during  each  phase  of 
the  acquisition  process.  Fifth,  establish  a  means  or  mechanism  for  collection, 
dissemination  and  utilization  of  such  documentation. 

8.  Identify  distribution  of  aptitudes  in  the  population  to  input  to  a 
DoD  requirements/skills  inventory  comparison  (e.g..  Project  TALENT). 

9.  Develop  Service-specific  instructions  for  Program  Managers  on  method¬ 
ology,  applications,  and  benefits  of  FEA,  and  include  in  curriculum  of  DoD 
Program  Manager's  Course  at  the  Defense  Systems  Management  College. 

10.  Obtain  information  on  how  FEAs  are  performed  in  other  countries. 


C 
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Appendix  B 


WORKING  GROUP  SUMMARIES 


The  workshop  participants  were  divided  into  three  working  groups.  Each 
working  group  was  to  discuss  the  following  six  issues  and  then  report  their 
conclusions  to  the  entire  workshop.  The  summary  reports  of  each  working 
group  are  on  the  following  pages. 


The  issues  discussed  were: 

1.  Characterize  what  an  effective,  practicable  FEA  technology  should 
be — including  a  definition  of  what  FEA  is,  what  the  essential 
components  of  FEA  are,  and  how  results  of  FEA  should  be  reported. 

2.  List  specific  technologies  we  have  that  can  be  used  now  to 
contribute  to  the  FEA  process. 

3.  List  specific  results/accomplishments  that  the  MP&T  community 
ought  to  achieve  in  the  next  2  years  in  support  of  FEA. 

A.  List  specific  results/accomplishments  that  the  MP&T  community 
ought  to  achieve  in  the  next  5-7  years  in  support  of  FEA. 

5.  Recommend  specific  follow-on  actions  that  should  occur  to 
produce  value  from  this  workshop. 

6.  Recommend  specific  actions  that  should  occur  to  establish  FEA  as 
an  essential  component  of  the  systems  acquisition  process. 


?. 


WORKING  GROUP  I 
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Issue  I.  FEA  Definition 


Steps  and  process  during  system  acquisition  cycle  which  are  required 
for  effective  and  timely  application  of  personnel  systems  and  training  tech¬ 
nologies  to  assure  the  trained  and  effective  human  resources  availability. 

Temperal  Perspective 

•  Up  to  Milestone  II 
Methods  and  Techniques 

•  Standardized  methodologies 
Components 


•  Requirements  of  the  system 

•  Manpower  requirements  (numbers,  spaces) 

•  Personnel  requirements  (faces,  skills) 

•  Training  capabilities  and  skills 

•  Selection  and  recruitment 

•  Costs  (acquisition,  recurring  and  facilities) 

Reporting  to  Project  Manager 

•  Manpower  and  personnel  resource  requirements  (spaces  and  faces) 

•  Training  requirements 

•  Equipment  and  facilities 

•  Costs 

•  Availability/obtainability 

•  Alternative  options 

•  Leverage  items:  Cost,  risk 

•  Constraints 

•  Tradeoffs 


Issue  2.  Manpower  Requirements 
Job/Task  Skill  Analysis 

•  Industrial  engineering  techniques 

•  Simulation  techniques 

Manpower  models 

•  Empirical  analysis  of  historical  data 

•  Force  planning/managing 

•  Topline  and  Topcat  (A/F  Personnel  Plan) 
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Training 

•  First  phase  training  analysis  (translation  of  skills/tasks  into 
training  requirements). 

•  Pre-job/task  analysis — Phase  I 

•  Post-job/task  analysis — Phase  II 

•  ISD 

•  TRAM3D — Determine  training  requirements  from  task  analysis 

Cost 

•  Accounting/budget  models 

•  LCC  models 

•  Econometric  models 

Availability 

Computerized  data  base 


Issues  3  &  4  (arranged  roughly  by  category,  with  rough  chronology  within 
category) 

Information  Acquisition  and  Dissemination 

•  Collect  and  publish  data  on  the  bad  effects,  including  higher  system 
costs,  of  neglecting  FEA  (horror  stories). 

•  Collect  and  publish  a  set  of  case  history  reports  on  early  and 
successful  implications  of  FEA  (fairy  tales). 

•  Develop  and  conduct  briefings  and  presentations  to  top  DoD  and 
congressional  people  demonstrating  cost  savings  and  improved  effectiveness 
resulting  from  FEA  methodology. 

•  Develop  and  publish  one  understandable,  condensed  guidebook  to  FEA 
for  program  managers  with  pointers  to  more  detailed  references  (menu). 

•  Develop  and  publish  an  understandable  workbook  for  FEA  for  practi¬ 
tioners  with  pointers  to  more  detailed  references  (cookbook). 

Methodology 

•  Work  toward  standardization/consol idation  of  analytical  techniques 
in  major  FEA  component  areas. 

•  Develop  lower  cost  model ing  techniques  for  key  FEA  factors. 

•  Develop  and  promote  availability  of  "universal"  computer  program 
packages  covering  frequently  used  FEA  techniques. 

•  Provide  introduction  to  analytical  computer  programs  in  DoD  program 
managers'  course. 

•  Develop  procedures  to  synthesize  quantitative  design  goals  for  the 

following: 
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a)  Desired  man-machine  performance 

b)  Manpower  requirements 

c)  Mastery  model  of  human  performance 

d)  Prerequisite  model  for  entering  learners 

e)  Learning  productivity  with  time 

f)  Cost  of  human  components 

•  Develop  guidelines  for  integration  of  estahl  islied  ISD  methodology 
with  total  FEA  process. 

•  Develop  guidelines  for  quality  assurance  procedures  on  FEA  compon¬ 
ents,  with  particular  emphasis  on  data  bases. 

•  Develop  methods  for  more  precise/quantitative  specification  of  system 
performance  factors  associated  with  the  human  component. 

•  Consider  new  visual  and  procedural  methods  for  front-end  analysis  aimed 
at  new  training  delivery  systems. 

Training  Systems  and  Alternatives 

•  Carefully  re-examine  concept  of  concurrent  procurement  of  final 
training  system  and  weapon  system — ID  and  evaluate  alternative  strategies. 

•  Develop  new  and  more  useful  principles  of  learning  transfer,  including 
negative  transfer  and  retention. 

•  Perform  more  careful  examination  of  capabilities  and  limitations  of 
embedded  training  as  alternative  training  method. 

•  Examine  and  evaluate  novel,  low-cost  means  of  training  as  alternatives 
to  both  high-fidelity  simulation  and  usage  of  operational  equipment  (e.g. , 
mental  rehearsal,  visual  imagry,  2-D  simulation,  gaming,  etc.). 

•  Develop  useful  and  agreed-upon  definitions  and  measures  of  training 
system  effectiveness  suitable  for  varied  stages  of  FEA  application. 

•  Develop  more  useful  categorization  of  training  media  and  technologies 
so  that  novel  training  alternatives  can  be  more  easily  identified. 

Background  Data  and  Definitions 

•  Identify  existing  data  base  that  contributes  to  early  definition  of 
training  systems,  publish  index  or  catalog. 

•  Establish  practical  data  base  for  those  pe rformance  factors  based  on 
current  deployment  of  reference  systems. 

•  Reassess  data  item  descriptions  and  generate  new  ones. 

•  Develop  a  more  general  metric  for  training  requirements  than  the  task. 
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•  Identify  the  critical  factors  (i.e. ,  drivers,  big  payoff  areas, 
levers,  etc.)  driving  results  oi  typical  FEA’s;  assess  state  of  data  bases 
relating  to  these  factors;  identify  lacks  in  scope  and  availability  of  these 
data  bases;  move  to  rectify  lack,  create  new  data  bases  where  necessary,  and 
to  develop  improved  methods  of  utilization. 

•  Develop  methodology  for  simple,  low-cost  means  of  generating  audit 
trails  during  the  course  0f  system  FEA;  establish  mechanism  for  collection, 
dissemination,  and  utilization  of  such  documentation. 


Issue  5 .  Follow-On  Actions 

•  Firm  up  methodology  of  reporting  to  Project  Managers. 

•  Implement  recommendations  of  Issues  3  and  4. 

•  Integrate  results  of  the  workshop  into  a  presentation  and  plan. 

•  Consolidate/prioritize  efforts  identified  in  Issues  2,  3  and  4. 

•  Have  a  follow-up  workshop  to  include  Project  Managers  and  their 
assessment  of  this  workshop. 


Issue  6.  Recommended  Actions 

•  Existing  directives  allow  for  training  FEA. 

•  Educate  Project  Managers  and  other  decision  makers  of  the  value 
impact  of  training  FEA  on  total  DoD  requirements.  Include  in  the  Curriculum 
of  Project  Manager's  school. 

•  Establish  an  advocacy  in  DoD  for  the  FEA  process. 

•  Have  the  advocate  take  steps  necessary  to  increase  R&D  funding  for 
FEA  on  a  timely  basis. 

•  Evaluate  adequacy  of  existing  data  base  and  methodology.  Identify 
deficiencies  and  establish  a  plan  to  correct  these  deficiencies. 

•  Assess  the  permanency  of  management  assignment  problem  as  it  applies 
to  continuity  of  FEA. 
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WORKING  GROUP  II 


Issue  1.  FEA  Definition 

FEA  is  a  threat-driven  needs  analysis.  It  is  an  ongoing  process  producing 
a  series  of  products.  It  is  initiated  at  a  pro-conceptual  stage  and  continuing 
on  through  Milestone  2.  Its  ultimate  goal  is  the  improvement  of  unit  readiness. 


Issue  2.  Specific  Technologies  Contributing  to  FEA 

•  Computer  simulation/modeling 

•  Batch  mode 

•  Man-in-the-loop  mode 

•  Mockups 

•  Static 

•  Dynamic 

•  ISD  (improved  training  system  approaches) 

•  Needs  refinement 

•  Falls  short  on  device/simulator/generic  system  selection 

•  COEA/CTEA 

•  Differentiate  cost :  COEA/CTEA 

Budget  cost  (line  14-16  in  PM  budget) 

•  Ancestral  system  history 

•  Parametric  projection 

•  Comparability  analysis 

•  Test  and  evaluation  tools 

•  Design  tests 

•  Operational  tests 

•  Unit  readiness  measures 

•  NTC 


Issue  3.  Accomplishments/Goals:  2  Year  Time  Frame 

•  Need  DoD  level  concentrated  briefing  on  the  outcome  of  this. 

•  Need  to  identify  JPA's  for  PM's. 

•  Need  to  have  LSA  data  requirements  laid  on  to  PM's. 

•  Need  "educational  program"  for  DoD  PM  School. 


Issue  4.  Accomplishroents/Goals :  5-7  Year  Time  Frame 

•  Need  skills  inventory/population  quality  census  (supply). 

•  Need  service  requireraents/skill  inventory 

•  Secretary  White's  August  1978  requirement  met  (demand). 

•  Need  "Supply-Demand"  comparator. 

•  Need  to  develop/design  a  Management  Information  System  (ARPANET- 1  ike) 
to  handle  these  data 

•  Tri-Service  contributed 

•  Tri-Service  accessible 

•  Identify  distribution  of  aptitudes  in  the  manpower  pool. 

•  "Reinitiate"  a  Project  Talent 

Issue  5.  Recommended  Follow-On  Actions 

•  Need  further  Tri-Service  meeting 

•  Six  months  hence 

•  Overlay  IHS  on  LCSMM  (Macro  I) 

•  Overlay  ISD  on  Macro  I  (Macro  II) 

•  Shred  out  Macro  II  to  more  refined  media  selection  requirement 

•  Develop  curriculum  for  PM  Course  at  Defense  System  Management  College, 
Ft.  Belvoir,  Virginia. 

•  Action  to  implement  curriculum  in  PM  School. 

•  Generation  of  Tri-Service  list  of  available  technologies. 

Issue  6.  Recommended  Actions  to  Insure  FEA  is  Essential  Component  of 
Systems  Acquisition  Process 

•  Assume  we  are  at  our  "milestone  zero."  Ignore  "old  ways" — implement 
new. 

•  Develop  more  valid  measures  of  unit  readiness  to  serve  as  "what  if" 
criteria  for  PM/TM  design  decisions. 

•  Implement  existing  regulations  (109,  5000.1,  .2,  etc.)  with  "teeth 
in  them." 

•  Have  FEA  community  examine  regulations  from  standpoint  of  providing 
PM' s  with  FEA  JPA’s. 

•  Accountability 

•  Implement  configuration  freeze  at  full  scale  design  review  stage. 
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WORKING  GROUP  III 


Issue  1.  Two  kinds  of  front-end  analyses  (FEA)  were  distinguished: 


1.  Concept  studies:  principally  designed  to  influence  the  design  of 
the  weapon  system.  These  are  a  part  of  and  will  contribute  to  the  sequence 
of  studies:  Weapon  systems  concepts  and  requirements  (e.g.,  MENS)  to  the 
Support  concepts  to  Personnel  performance  requirements  to  Training  subsystem 
requirements. 


Weapon  Systems  (e.g.,  MENS) 

Support  Concepts 

Personnel  Performance  Requirements 
Training  Subsystem  Requirements 


2.  Training  studies:  principally  designed  to  influence  the  develop¬ 
ment  of  the  training  subsystem  including  such  models  as  the  1SD  framework. 

The  Products  of  these  studies  will  generally  be  requirements  statements. 
The  specificity  and  depth  of  the  analysis  will  depend  upon  when  in  the  weapon 
systems  development  cycle  the  study  is  done.  In  the  early  stages,  the 
analysis  will  be  gross  and  general;  with  iteration,  details  and  depth  will 
evolve. 

There  was  no  agreement  on  a  definition  of  FEA.  There  was  even  some 
question  as  to  whether  FEA  could  be  defined.  However,  there  was  agreement 
that  certain  kinds  of  analytic  products  had  to  be  generated  at  specific  times. 


Issue  2.  The  following  technologies  were  noted  but  with  primary  emphasis 
on  how  they  could  be  improved: 

•  Data  to  define  critical  training  parameters. 

•  Identification  of  numbers  and  kinds  of  people  and  associated  skills. 

•  Development  of  handbooks  and  guides. 

•  Manpower,  Personnel,  and  Training  analyses  techniques. 

•  Prediction  techniques  for  future  manpower,  personnel  and  training  pool 

•  Techniques  for  early  communication  with  designers. 

•  Ways  of  defining  what  people  can  really  do  rather  than  labels. 

•  Improvements  in  cost  and  performance  data  bases. 

•  Guides  for  engineers  on  automation. 


A  basic  point  was  better  use  of  the  data  already  available.  The 
present  "data  base"  is  not  properly  accumulated,  assimilated  and  distributed. 


DATA  AVAILABLE 


APPLICATIONS 


IMPROVED  DATA 


NEW  DATA 


We  use  our  present  technology  base  poorly;  we  need  to  spend  more  time  and 
effort  on  getting  better  processing  of  what  we  already  know. 


Issue  3.  Next  Two  Years 


•  Develop  methods  by  which  users  can  better  identify  training  needs. 

•  New  methods  for  skill  level  determination. 

0  Further  work  on  prototype  models  for  MP&T  analyses. 

•  New  methods  for  aggregation  of  MP&T  demands  and  supply  for  future 
systems. 

•  Better  cost  data,  better  cost  models  and  examples  of  how  to  use 
cost  data  and  cost  models. 

•  Development  of  technical  communication  systems  between  management, 
users,  designers,  and  R&D  communities  for  MP&T. 

•  Consolidation  of  existing  training  data  for  better  use. 

•  Operational  definitions  of  inputs,  processes,  and  outputs  of  FEA. 

•  Common  analysis  data  bases  that  can  be  used  by  all  Services. 

•  Using  cost-effectiveness  data  apply  to  fundamental  system  feasibility 
studies  as  well  as  fine  tuning  systems. 

THE  MAJOR  SYSTEM  EMPHASIS  FOR  THIS  GROUP  WAS  MAINTENANCE  AND  MAINTAINABILITY 
AND  MAINTENANCE  TRAINING.  GOAL:  80%  IMPROVEMENT  IN  MAINTENANCE  PERFORMANCE. 
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Issue  4.  Next  5-7  Years 


•  Extend  user  training  need  identification  system 

•  Continue  work  on  aggregation  manpower  models. 

•  Continue  work  on  handbook  and  guides. 

•  Further  development  of  linkage  efforts:  MANAGEMENT-USERS-DESIGNERS-R&D 

•  More  meaningful  analysis  of  the  R&D  community. 

•  Question  assumptions  about  training. 

•  Observe  and  record  the  validity  of  MP&T  predictions. 

•  Improved  productivity  potential  of  100%  in  maintenance  performance. 

•  Continue  development  of  common  analysis  data  base. 

•  Continue  development  of  computerized  models  for  MP&T  in  system  design. 

•  Conduct  field  demonstrations  of  different  maintenance  concepts. 

•  Validate  the  front-end  process  in  itself. 


Issue  5 .  Specific  Follow-On  Actions 

•  Identify  an  OSD  Action  Officer  (UNANIMOUS  VOTE). 

•  Begin  to  standardize  terms. 

•  Begin  to  identify  the  real  technology  data  base. 

•  Begin  to  institute  the  management-user-designer  dialogue. 

•  Look  for  new  forums  for  these  problems  (e.g. ,  the  TAGs). 

•  Appoint  small  working  group  to  continue  work. 

•  A  report  from  Dr.  Fletcher  and  CDR  Chatelier  on  what  they  heard 
in  this  workshop:  good  and  bad. 


Issue  6.  Action  to  Establish  FEA 

•  Develop  incentives  to  contractors  for  doing  FEA. 

•  Develop  job  aids  on  FEA:  How  should  they  be  done?  How  described  in 
SOW? 

•  Keep  all  informed  on  MP&T  policy  for  acquisition  (e.g..  Project 
HARDMAN)  . 


